System and method of controlling external apparatus connected with device

ABSTRACT

A mobile device includes a memory configured to store a program; and a processor configured to control one or more external apparatuses by executing the program. The program includes commands which, when executed by the processor, cause the processor to identify the one or more external apparatuses communicable with and controllable by the mobile device, from a plurality of external apparatuses; provide apparatus information of the one or more external apparatuses, to a server; receive, from the server, control information for controlling the one or more external apparatuses for a user&#39;s intention; and transmit a control command to the one or more external apparatuses based on the received control information.

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/679,749, filed Apr. 6, 2015, which claims the benefit of U.S.Provisional Application No. 61/975,187, filed Apr. 4, 2014, and is acontinuation-in-part of U.S. patent application Ser. No. 14/022,556,filed Sep. 10, 2013, which claims priority from Korean PatentApplication No. 10-2012-0099739, filed Sep. 10, 2012, in the KoreanIntellectual Property Office. The disclosures of the above-namedapplications are incorporated herein in their entireties by reference.

BACKGROUND 1. Field

Apparatuses and methods consistent with exemplary embodiments relate tocontrolling an external apparatus connected with a device according to auser's intention.

2. Description of the Related Art

Due to development of natural language analysis technologies and mobiletechnologies, a device may analyze information input by a user and mayoperate according to the user's intention. However, even though a devicemay be able to operate according to a user's intention, it is difficultfor the device to utilize various connectable or device-compliantapparatuses. As such, it is difficult to control a device by effectivelyreflecting various true intentions of a user.

Accordingly, there is a need for methods and apparatuses for obtainingvarious types of condition information by using a device and an externalapparatus connectable with the device and for operating the device anddifferent external apparatuses by effectively reflecting variousintentions of a user.

SUMMARY

Exemplary embodiments may address at least the above problems and/ordisadvantages and other disadvantages not described above. The exemplaryembodiments are not required to overcome the disadvantages describedabove, and an exemplary embodiment may not overcome any of the problemsdescribed above.

One or more of exemplary embodiments provide a system and method ofcontrolling an external apparatus connected with a device by determininga user's intention based on user input information, and controlling thedevice and the external apparatus according to the user's intention.

According to an aspect of an exemplary embodiment, there is provided aserver configured to control at least one external apparatus, the serverincluding: a communicator configured to receive an input command of auser and biometric information of the user from a device; and acontroller configured to determine a condition for controlling the atleast one external apparatus and operation of the at least one externalapparatus based on the received input command of the user and thereceived biometric information of the user, and provide a controlcommand for controlling the operation of the at least one externalapparatus to the at least one external apparatus if the determinedcondition is satisfied.

According to another aspect of an exemplary embodiment, there isprovided a method of controlling at least one external apparatus, themethod including: receiving an input command of a user and biometricinformation of the user from a device; and determining a condition forcontrolling the at least one external apparatus and operation of the atleast one external apparatus based on the received input command of theuser and the received biometric information of the user; and providing acontrol command for controlling the operation of the at least oneexternal apparatus to the at least one external apparatus if thedetermined condition is satisfied.

According to another aspect of an exemplary embodiment, there isprovided a server configured to control at least one external apparatus,the server including: a communicator configured to receive an inputcommand of a user and biometric information of the user from a device;and a controller configured to determine a condition for controlling theat least one external apparatus and operation of the at least oneexternal apparatus based on the received input command of the user andthe received biometric information of the user, and provide a controlcommand for controlling the operation of the at least one externalapparatus to the at least one external apparatus if the determinedcondition is satisfied.

According to another aspect of an exemplary embodiment, there isprovided a method of controlling at least one external apparatus, themethod including: receiving an input command of a user and biometricinformation of the user from a device; and determining a condition forcontrolling the at least one external apparatus and operation of the atleast one external apparatus based on the received input command of theuser and the received biometric information of the user; and providing acontrol command for controlling the operation of the at least oneexternal apparatus to the at least one external apparatus if thedetermined condition is satisfied.

According to another aspect of an exemplary embodiment, there isprovided a non-transitory computer-readable recording medium havingrecorded thereon a computer program which, when executed by a computersystem, performs the method.

According to another aspect of an exemplary embodiment, there isprovided a non-transitory computer-readable recording medium havingrecorded thereon a computer program which, when executed by a computersystem, performs the method.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and/or other aspects will become more apparent by describingcertain exemplary embodiments, with reference to the accompanyingdrawings, in which:

FIG. 1 is an overall schematic diagram of a system for controlling anexternal apparatus via a device, according to an exemplary embodiment;

FIG. 2 is a flowchart of a method of controlling an external apparatusby a device, according to an exemplary embodiment;

FIG. 3 is a flowchart of a method of controlling an external apparatusby a device, according to an exemplary embodiment;

FIG. 4 is a flowchart of a method of controlling an external apparatusby a device, according to an exemplary embodiment;

FIG. 5 is a flowchart of a method of controlling an external apparatusby a device, according to an exemplary embodiment;

FIG. 6 is an operation table, according to an exemplary embodiment;

FIG. 7 is a schematic diagram of controlling an external apparatus,according to an exemplary embodiment;

FIG. 8 is a schematic diagram of controlling an external apparatus,according to an exemplary embodiment;

FIG. 9 is a schematic diagram of controlling an external apparatus,according to an exemplary embodiment;

FIG. 10 is a schematic diagram of controlling an external apparatusbased on a voice command and biometric information of a user, which isperformed by a server, according to one or more exemplary embodiments;

FIG. 11 is a detailed flowchart of a method of controlling an externalapparatus according to a voice command of a user in consideration ofbiometric information of the user, which is performed by a server,according to one or more exemplary embodiments;

FIG. 12 is a schematic diagram showing an example of an operation tableof a device for controlling an external apparatus based on biometricinformation, according to one or more exemplary embodiments;

FIG. 13 is a schematic diagram showing an example of controlling anexternal apparatus based on a voice command and biometric information ofa user, which is performed by a server, according to one or moreexemplary embodiments;

FIG. 14 is a flowchart of a method of controlling the external apparatusaccording to a voice command of a user in consideration of healthinformation about the user, which is performed by a server, according toone or more exemplary embodiments;

FIG. 15 is a schematic diagram showing an example of an operation tableof a device for controlling an external apparatus based on a healthstate of a user, according to one or more exemplary embodiments;

FIGS. 16 and 17 are schematic diagrams showing an example of receiving avoice command of a user, which is performed by a device, according toone or more exemplary embodiments;

FIGS. 18, 19, 20, 21, and 22 are schematic diagrams showing an exampleof receiving a user input of health information of a user, which isperformed by a device, according to one or more exemplary embodiments;

FIG. 23 is a block diagram of a device according to an exemplaryembodiment;

FIG. 24 is a block diagram of a condition information obtainer accordingto an exemplary embodiment;

FIG. 25 is a block diagram of a device according to another exemplaryembodiment; and

FIG. 26 is a block diagram of a server according to an exemplaryembodiment.

DETAILED DESCRIPTION

Certain exemplary embodiments are described in greater detail below withreference to the accompanying drawings.

In the following description, the same drawing reference numerals areused for the same elements even in different drawings. The mattersdefined in the description, such as detailed construction and elements,are provided to assist in a comprehensive understanding of exemplaryembodiments. Thus, it is apparent that exemplary embodiments can becarried out without those specifically defined matters. Also, well-knownfunctions or constructions are not described in detail since they wouldobscure exemplary embodiments with unnecessary detail.

Throughout the specification, it should be understood that when anelement is referred to as being “connected with” another element, it canbe directly connected with the other element, or electrically connectedwith the other element while intervening elements may also be present.Also, when a part “includes” or “comprises” an element, unless there isa particular description contrary thereto, the part can further includeother elements, not excluding the other elements.

Expressions such as “at least one of,” when preceding a list ofelements, modify the entire list of elements and do not modify theindividual elements of the list.

FIG. 1 is an overall schematic diagram of a system for controlling anexternal apparatus via a device, according to an exemplary embodiment.

As illustrated in FIG. 1, the system includes the device 1000, one ormore external apparatuses 2000 connected with the device 1000, a server3000, and a network 4002.

Referring to FIG. 1, the device 1000 may generate user input informationbased on a user input, and may provide the user input information,apparatus information of the external apparatus 2000, and conditioninformation to the server 3000 such that the external apparatus 2000 maybe controlled according to a user's intention.

The device 1000 may receive a text input or a voice input of the user,may generate the user input information based on the received input, andmay provide the generated user input information to the server 3000. Thedevice 1000 may check the external apparatus 2000 connectable with thedevice 1000, and may obtain and provide the apparatus information of theexternal apparatus 2000 to the server 3000. Furthermore, the device 1000may obtain the condition information via the device 1000 or the externalapparatus 2000, and may provide the obtained condition information tothe server 3000. The device 1000 may obtain the condition informationbased on control information to be described below, but is not limitedthereto.

The server 3000 may analyze the user's intention based on the user inputinformation, and may generate and provide the control information to thedevice 1000 such that the device 1000 may operate according to theuser's intention and that the external apparatus 2000 may be controlledby the device 1000. The server 3000 may determine the user's intentionby analyzing the user input information, and may determine a conditionfor controlling the external apparatus 2000 in such a way that theuser's intention is satisfied. The server 3000 may check whether thedetermined condition is satisfied and may generate the controlinformation for controlling operations of the device 1000 and theexternal apparatus 2000.

The device 1000 may receive the control information from the server3000, and may transmit a control command to some external apparatuses2000 based on the control information.

The device 1000 is an apparatus capable of transmitting and receivinginformation to and from the server 3000 and of transmitting the controlcommand to the external apparatus 2000, and may be, for example, asmartphone, a mobile phone, a personal digital assistant (PDA), a laptopcomputer, a media player, a global positioning system (GPS) apparatus,or another mobile or non-mobile computing apparatus, but is not limitedthereto.

The external apparatus 2000 may be an apparatus connectable with andcontrollable by the device 1000, and/or capable of at least partiallysharing functions provided by the device 1000. The external apparatus2000 may be a stand-alone apparatus capable of independently executingan application, but is not limited thereto. The external apparatus 2000may include an input apparatus, an output apparatus, or a controlapparatus, for example, a home appliance such as a smart TV or an airconditioner, a security camera, a recorder, a mobile phone, a personalcomputer (PC), a pillow vibrator, a motor vehicle, a navigator, amicrophone, a speaker, a pedal, a joystick, a musical instrument (e.g.,a piano, an organ, an electronic keyboard, a guitar, a violin, or acello), a game manipulator, a doll, a medical appliance, sportingequipment, a camera, or a sensor.

The server 3000 may be a service providing server for providing thecontrol information for allowing the device 1000 to control the externalapparatus 2000 according to the user's intention, but is not limitedthereto.

The server 3000 may be a cloud server, for example, a PC of the user. Ifthe server 3000 is a cloud server, the server 3000 may include apredetermined intelligence engine, may analyze the user's intention viathe intelligence engine, and may transmit the control information forcontrolling the external apparatus 2000 to the device 1000. Otherwise,if the server 3000 is a client server, the server 3000 may receivepredetermined information from a separate service providing server(e.g., a weather information providing server or a traffic informationproviding server).

The network 4002 may be implemented as a wired network such as a localarea network (LAN), a wide area network (WAN), or a value added network(VAN), or a wireless network such as a mobile radio communicationnetwork, a near field communication (NFC) network, or a satellitecommunication network. The network 4002 may be a comprehensive datacommunication network for allowing appropriate communications betweenthe network components illustrated in FIG. 1, and may include wiredInternet, wireless Internet, and/or a mobile wireless communicationnetwork.

FIG. 2 is a flowchart of a method of controlling the external apparatus2000 by the device 1000 based on control information generated by theserver 3000, according to an exemplary embodiment.

In operation S200, the device 1000 checks the external apparatus 2000controllable by the device 1000. The device 1000 may check the externalapparatus 2000 controllable by the device 1000 from among the externalapparatuses 2000 connectable with the device 1000. The controllableexternal apparatus 2000 may be an external apparatus connected orconnectable with the device 1000 and having installed therein apredetermined application for utilizing a control service according toan exemplary embodiment. If the application is not installed in thecontrollable external apparatus 2000, the device 1000 may performcommunication connection and pairing with the external apparatus 2000,and may provide to the external apparatus 2000 the application or linkinformation for downloading the application.

The external apparatus 2000 may include, for example, a home appliancesuch as a smart TV or an air conditioner, a security camera, a recorder,a mobile phone, a PC, a pillow vibrator, a motor vehicle, or anavigator, but is not limited thereto. The device 1000 and the externalapparatus 2000 may be connected with each other via a wired network suchas a LAN, a WAN, or a VAN, or a wireless network such as a mobile radiocommunication network, an NFC network, or a satellite communicationnetwork. The device 1000 may be connected with the external apparatus2000 via, for example, a home gateway (not shown).

The device 1000 may receive from the checked external apparatus 2000apparatus information of the external apparatus 2000, but is not limitedthereto. The device 1000 may receive the apparatus information of theexternal apparatus 2000 from a separate server (not shown). For example,if the device 1000 is connected with the external apparatus 2000 via ahome gateway (not shown), the device 1000 may request the home gatewayfor the apparatus information of the external apparatus 2000. Theapparatus information of the external apparatus 2000 may includeinformation about at least one of an identification value of theexternal apparatus 2000, a media access control (MAC) address, a serviceset identifier (SSID), the type of the external apparatus 2000,capability provided by the external apparatus 2000, a category, and acommand used to control the external apparatus 2000. The capabilityprovided by the external apparatus 2000 may include, for example, avoice output capability, a video output capability, a voice recordingcapability, an image capturing capability, and a humidity sensingcapability, but is not limited thereto.

After user input information is generated in operation 5202 or a user'sintention is determined in operation S204, the device 1000 may searchfor the external apparatus 2000 to be controlled by the device 1000,based on the user input information or the user's intention.

In operation 5202, the device 1000 provides the user input informationto the server 3000. The device 1000 may generate the user inputinformation based on a user input, and may provide the generated inputinformation to the server 3000. The user input information may beinformation about at least one of text input by the user, a voice of theuser, a facial expression of the user, a gesture of the user, and aphysical state of the user.

If the user inputs text to the device 1000, the device 1000 may providethe text data input by the user as the user input information to theserver 3000. For example, if the user inputs text such as “Wake me up at7:00 if it doesn't rain tomorrow” to the device 1000, the device 1000may provide the input text data as the user input information to theserver 3000. Otherwise, if the user inputs a voice to the device 1000,the device 1000 may provide the input voice data as the user inputinformation to the server 3000. The device 1000 may transform the inputvoice data into text data, and may transmit the transformed text data asthe user input information to the server 3000. Alternatively, forexample, the device 1000 may generate the user input information byusing an email or a text message stored in the device 1000.

The device 1000 may parse the user input information, and may transmitthe parsed data to the server 3000. If the user input information istext data, the device 1000 may parse the text data. Otherwise, if theuser input information is voice data, the device 1000 may transform thevoice data into text data and may parse the transformed text data.However, an exemplary embodiment is not limited thereto. The device 1000may provide a voice data file to the server 3000, and the server 3000may transform and parse the received voice data file into text.

In operation S204, the server 3000 determines the user's intention basedon the user input information. The server 3000 may determine the user'sintention by analyzing the user input information. In order to analyzereceived text data as the user input information, the server 3000 mayuse various natural language analysis methods. For example, the server3000 may analyze the text data by using natural language processing suchas morpheme analysis, syntax analysis, or named entity recognition.Alternatively, the server 3000 may determine the user's intention basedon the user input information by using, for example, ontology-basedreasoning and probability-based reasoning. If the server 3000 receivesvoice data as the user input information, the server 3000 may transformthe received voice data into text data, and may analyze the transformedtext data. The server 3000 may generate user intention information byanalyzing the user input information. For example, if the user inputinformation is “Wake me up at 7:00 if it doesn't rain tomorrow”, theserver 3000 may generate the user intention information including timeinformation such as “tomorrow, 7:00”, information related to conditioninformation such as “if it doesn't rain”, and operation information suchas “wake me up”. The information related to condition information may beinformation used by the device 1000 or the server 3000 to obtaincondition information about the user's intention.

Alternatively, the server 3000 may determine the user's intention via aninteractive interface using feedback. For example, if the user'sintention is not easily determined based on the user input informationreceived from the device 1000, the server 3000 may request the device1000 for additional user input information. The server 3000 may notifyadditionally needed information to the device 1000. The server 3000 maydetermine the user's intention by using the user input informationadditionally received from the device 1000. For example, if user inputinformation such as “Give me a call if the baby wakes up” is receivedfrom the device 1000, the server 3000 may request the device 1000 toadditionally provide information about a point of time when the babywakes up. The device 1000 may additionally provide user inputinformation such as “Give me a call if the baby wakes up tomorrowmorning” to the server 3000, and the server 3000 may determine theuser's intention based on the additionally received user inputinformation.

In operation S206, the device 1000 provides the apparatus information ofthe checked external apparatus 2000 to the server 3000. The device 1000may provide information about at least one of an identification value ofthe external apparatus 2000, a MAC address, an SSID, the type of theexternal apparatus 2000, capability provided by the external apparatus2000, a category, and a command used to control the external apparatus2000 to the server 3000. In operation S206, the device 1000 may providethe apparatus information of the device 1000 to the server 3000. Theapparatus information of the device 1000 may include information aboutat least one of, for example, an identification value of the device1000, a MAC address, an SSID, the type of the device 1000, capabilityprovided by the device 1000, a category, and a command used to controlthe device 1000.

In operation S208, the server 3000 determines a condition forcontrolling the external apparatus 2000 based on the user's intention.The condition for controlling the external apparatus 2000 is a conditionfor allowing a user-desired operation to be performed according to theuser's intention. For example, if the user input information is “Wake meup if it doesn't rain at 7:00 tomorrow”, a condition such as “if itdoesn't rain at 7:00 tomorrow” for a user-desired operation such as“wake me up” may be the condition for controlling the external apparatus2000. The server 3000 may determine a condition that has to be satisfiedto control operation of the external apparatus 2000 or the device 1000,based on the user intention information. For example, if the userintention information includes time information such as “tomorrow, 7:00”and information related to condition information such as “if it doesn'train”, the server 3000 may determine a time condition such as“tomorrow”, a weather condition such as “sunny, cloudy”, and a placecondition such as “device location” as the condition for controlling theexternal apparatus 2000. In order to ensure that the determinedcondition is satisfied, a predetermined external apparatus 2000 may becontrolled.

In operation S210, the server 3000 selects the external apparatus 2000related to the user's intention. The server 3000 may select the externalapparatus 2000 related to the user's intention, based on operationinformation included in the user intention information and the apparatusinformation received from the device 1000. For example, if the userintention information includes operation information such as “wake meup”, the server 3000 may select the external apparatus 2000 to wake upthe user, by using the apparatus information received from the device1000. The server 3000 may select, for example, a pillow vibrator and/ora lighting device as the external apparatuses 2000 for waking up theuser.

In operation S212, the server 3000 generates control information forsatisfying the user's intention. The control information may includeinformation for checking whether the condition determined by the server3000 is satisfied, and information for controlling operation of at leastone of the device 1000 and the selected external apparatus 2000 if thecondition determined by the server 3000 is satisfied. The controlinformation may include a control command for controlling operation ofthe device 1000 or the external apparatus 2000, and information togenerate the control command.

In more detail, the server 3000 may generate the control information forchecking whether the condition determined by the server 3000 issatisfied, based on the condition determined by the server 3000 and theapparatus information received from the device 1000. For example, if thecondition determined by the server 3000 includes a time condition suchas “tomorrow”, a weather condition such as “sunny, cloudy”, and a placecondition such as “device location”, the server 3000 may generate thecontrol information for checking whether the condition determined by theserver 3000 is satisfied by allowing the device 1000 to check the date,the location of the device 1000, and the weather via the device 1000 orthe external apparatus 2000. The control information may include acontrol command instructing to check the date and the weather, a controlcommand instructing to obtain humidity information from a humiditysensor, and a command instructing to check the location of the device1000, and the control commands included in the control information maybe aligned in a predetermined order.

However, the control information is not limited thereto and may omit acontrol command but may include information referred to so as togenerate a control command, for example, “determine whether the weatheris sunny or cloudy at a device location tomorrow”.

If the condition determined by the server 3000 is satisfied, the server3000 may generate the control information for operating the device 1000or the external apparatus 2000. The server 3000 may generate the controlinformation for operating the device 1000 or the external apparatus2000, based on the operation information included in the user intentioninformation and the apparatus information received from the device 1000.For example, if the user intention information includes operationinformation such as “wake me up”, the server 3000 may generate a controlcommand instructing the device 1000 to output alarm information, acontrol command instructing a lighting device to be turned on, and/or acontrol command instructing a pillow vibrator to generate vibration.

The control information may include a plurality of control commands forcontrolling operation of at least one of the device 1000 and theexternal apparatus 2000, and the plurality of control commands may beexecuted in a predetermined order by at least one of the device 1000 andthe external apparatus 2000.

In operation 5214, the server 3000 provides the generated controlinformation to the device 1000.

In operation 5216, the device 1000 checks whether the condition forcontrolling the external apparatus 2000 is satisfied. The device 1000may check whether the condition determined by the server 3000 issatisfied, based on the control information received from the server3000. If the control information received from the server 3000 includesa control command for checking whether the condition for controlling theexternal apparatus 2000 is satisfied, the device 1000 may operateaccording to the control command included in the control information tocheck whether the condition is satisfied. Otherwise, if the controlinformation received from the server 3000 does not include the controlcommand for checking whether the condition for controlling the externalapparatus 2000 is satisfied, the device 1000 may generate a controlcommand based on the control information, and may check whether thecondition is satisfied, according to the generated control command.

In operation 5218, the device 1000 provides the control command to theexternal apparatus 2000 based on the control information. The device1000 may transmit the control command to the external apparatus 2000 inorder to follow the user's intention if the condition for controllingthe external apparatus 2000 is satisfied. If the condition forcontrolling the external apparatus 2000 is satisfied, the device 1000may transmit the control commands for operating the external apparatuses2000 to the external apparatuses 2000 in a predetermined order. Thedevice 1000 may perform a predetermined operation based on the controlinformation.

In operation 5220, another external apparatus (not shown) isadditionally connected with the device 1000 and, in operation 5224, thedevice 1000 provides apparatus information of the additionally connectedexternal apparatus to the server 3000.

In operation 5226, the server 3000 generates control informationincluding a control command about the additionally connected externalapparatus. The server 3000 may change the control information forsatisfying the user's intention, based on the apparatus information ofthe additionally connected external apparatus.

In operation 5228, the server 3000 provides the changed controlinformation to the device 1000. As such, the device 1000 may controloperation of the device 1000 or the external apparatus 2000 based on thechanged control information.

FIG. 3 is a flowchart of a method of controlling the external apparatus2000 by the device 1000 based on control information generated by theserver 3000, according to an exemplary embodiment. In FIG. 3, the server3000 may check whether a condition for controlling the externalapparatus 2000 is satisfied. Since some of the operations of FIG. 3 aresimilar to those described above with reference to FIG. 2, the detaileddescription thereof is omitted.

In operation S300, the device 1000 checks the external apparatus 2000controllable by the device 1000.

In operation S302, the device 1000 provides user input information andthe apparatus information to the server 3000. The device 1000 maygenerate the user input information based on a user input, and mayprovide the generated input information to the server 3000.

Also, the device 1000 may provide information about at least one of anidentification value of the external apparatus 2000, the type of theexternal apparatus 2000, and a command used to control the externalapparatus 2000 to the server 3000.

In operation S304, the server 3000 determines the user's intention basedon the user input information. The server 3000 may determine the user'sintention by analyzing the user input information. In order to analyzereceived text data as the user input information, the server 3000 mayuse various natural language analysis methods. For example, the server3000 may analyze the text data by using natural language processing suchas morpheme analysis, syntax analysis, or named entity recognition. Ifthe server 3000 receives voice data as the user input information, theserver 3000 may transform the received voice data into text data, andmay analyze the transformed text data. Also, the server 3000 maygenerate user intention information by analyzing the user inputinformation.

In operation S306, the server 3000 determines a condition forcontrolling the external apparatus 2000 based on the user's intention.The server 3000 may determine a condition that has to be satisfied tocontrol operation of the external apparatus 2000 or the device 1000,based on the user intention information. For example, if the userintention information includes time information such as “tomorrow, 7:00”and information related to condition information such as “if it doesn'train”, the server 3000 may determine a time condition such as“tomorrow”, a weather condition such as “sunny, cloudy”, and a placecondition such as “device location” as the condition for controlling theexternal apparatus 2000.

In operation S308, the server 3000 requests the device 1000 forcondition information. The server 3000 may request the device 1000 forthe condition information obtainable by the device 1000 and thecondition information obtainable by the external apparatus 2000, basedon the condition information. The server 3000 may request the device1000 for the condition information needed to check whether thedetermined condition is satisfied. For example, if a time condition, aweather condition, and a place condition are determined, the server 3000may request the device 1000 to provide information about the weather ata fixed time at a fixed place as the condition information to the server3000. The server 3000 may provide the type of the condition informationto the device 1000, but is not limited thereto. The server 3000 mayprovide to the device 1000 a control command instructing the device 1000and the external apparatus 2000 to obtain the condition information.

In operation S310, the device 1000 obtains the condition information.The device 1000 may obtain the condition information in response to arequest to obtain the condition information. The device 1000 may obtainvarious types of the condition information from the external apparatus2000 and a separate server (not shown). For example, the device 1000 mayobtain location information of the device 1000 from a locationinformation providing server (not shown), may obtain weather informationaround the device 1000 from a weather server (not shown), and may obtaintraffic information about a predetermined route from a trafficinformation providing server (not shown).

Also, for example, the device 1000 may obtain humidity information froma humidity sensor (not shown), may obtain a captured image from aphotographing apparatus (not shown), and may obtain recorded voice datafrom a recording apparatus (not shown). The device 1000 may obtain thecondition information from an apparatus included in the device 1000. Forexample, the device 1000 may obtain various types of the conditioninformation from a temperature sensor, a humidity sensor, a geomagneticsensor, a gravity sensor, a motion sensor, a gyro sensor, a camera, anda recorder included in the device 1000. The device 1000 may obtain thecondition information about a physical state of the user. The conditioninformation about the physical state of the user may be, for example,information about a heart rate, an electrocardiogram (ECG), breathing, apulse, and physical exercise of the user, and may be obtained from thedevice 1000 or the external apparatus 2000 connected with the device1000. Furthermore, the device 1000 may obtain the condition informationabout a network status, a login status of the external apparatus 2000,an application installed in the device 1000 or the external apparatus2000, and a user using the application. However, the device 1000 is notlimited thereto and may obtain various types of information as thecondition information.

In operation S312, the device 1000 provides the obtained conditioninformation to the server 3000. The device 1000 may obtain the conditioninformation in a preset cycle, and may provide the obtained conditioninformation to the server 3000 in real time, but is not limited thereto.The server 3000 may obtain the condition information from another device(not shown) or another server (not shown).

In operation S314, the server 3000 checks whether the condition forcontrolling the external apparatus 2000 is satisfied, based on thecondition information. The server 3000 may check whether the conditionis satisfied, based on the condition information received from at leastone of the device 1000, the other device, and the other server.

In operation S316, the server 3000 selects the external apparatus 2000related to the user's intention. The server 3000 may check an operationfor satisfying the user's intention, and may select the externalapparatus 2000 for performing the checked operation. For example, if theuser intention information includes operation information such as “wakeme up”, the server 3000 may select a lighting device and/or a pillowvibrator.

In operation S318, the server 3000 generates control information aboutthe selected external apparatus 2000. If the condition determined by theserver 3000 is satisfied, the server 3000 may generate the controlinformation for operating the device 1000 or the external apparatus2000. The server 3000 may generate the control information for operatingthe device 1000 or the external apparatus 2000, based on the operationinformation included in the user intention information and the apparatusinformation received from the device 1000. For example, if the userintention information includes operation information such as “wake meup”, the server 3000 may generate a control command instructing thedevice 1000 to output alarm information, a control command instructing alighting device to be turned on, and a control command instructing apillow vibrator to generate vibration. The control commands included inthe control information generated by the server 3000 may be aligned in apredetermined order.

In operation S320, the server 3000 provides the generated controlinformation to the device 1000.

In operation S322, the device 1000 transmits the control command to theexternal apparatus 2000 based on the control information. The device1000 may transmit the control commands for operating the externalapparatuses 2000 to the external apparatuses 2000 in a predeterminedorder. The device 1000 may perform a predetermined operation based onthe control information.

In operation 5324, another external apparatus (not shown) isadditionally connected with the device 1000 and, in operation 5326, thedevice 1000 provides apparatus information of the additionally connectedexternal apparatus to the server 3000.

In operation 5328, the server 3000 generates control informationincluding a control command about the additionally connected externalapparatus. The server 3000 may change the control information forsatisfying the user's intention, based on the apparatus information ofthe additionally connected external apparatus.

In operation 5330, the server 3000 provides the changed controlinformation to the device 1000. As such, the device 1000 may controloperation of the device 1000 or the external apparatus 2000 based on thechanged control information.

FIG. 4 is a flowchart of a method of generating a control command andcontrolling the external apparatus 2000 by the device 1000 based on userintention information received from the server 3000, according to anexemplary embodiment. In FIG. 4, the device 1000 may determine acondition for controlling the external apparatus 2000, based on the userintention information. Since some of the operations of FIG. 4 aresimilar to those described above with reference to FIGS. 2 and 3, thedetailed description thereof is omitted.

In operation 5400, the device 1000 checks the external apparatus 2000controllable by the device 1000 and, in operation 5402, the device 1000provides user input information to the server 3000. In operation 5404,the server 3000 determines a user's intention based on the user inputinformation. Operations 5400, 5402, and 5404 may respectively correspondto operations S200, 5202, and S204.

In operation 5406, the server 3000 provides user intention informationto the device 1000. For example, if the user input information is “Wakeme up at 7:00 if it doesn't rain tomorrow”, the server 3000 may provideto the device 1000 the user intention information including timeinformation such as “tomorrow, 7:00”, condition information such as “ifit doesn't rain”, and operation information such as “wake me up”.

In operation 5408, the device 1000 determines a condition forcontrolling the external apparatus 2000 based on the user's intention.The device 1000 may decide a condition that has to be satisfied tocontrol operation of the external apparatus 2000 or the device 1000,based on the user intention information. For example, if the userintention information includes time information such as “tomorrow, 7:00”and condition information such as “if it doesn't rain”, the device 1000may decide a time condition such as “tomorrow”, a weather condition suchas “sunny, cloudy”, and a place condition such as “device location” asthe condition for controlling the external apparatus 2000.

In operation S410, the device 1000 obtains condition information. Thedevice 1000 may obtain various types of the condition information fromthe external apparatus 2000 and a separate server (not shown).

In operation S412, the device 1000 obtains apparatus information of theexternal apparatus 2000. The device 1000 may receive the apparatusinformation of the external apparatus 2000 from the external apparatus2000, but is not limited thereto. The device 1000 may receive theapparatus information of the external apparatus 2000 from a separateserver (not shown). For example, if the device 1000 is connected withthe external apparatus 2000 via a home gateway (not shown), the device1000 may request the home gateway for the apparatus information of theexternal apparatus 2000. The apparatus information of the externalapparatus 2000 may include information about at least one of anidentification value of the external apparatus 2000, the type of theexternal apparatus 2000, and a command used to control the externalapparatus 2000.

In operation S414, the device 1000 selects the external apparatus 2000related to the user's intention. The device 1000 may check an operationfor satisfying the user's intention, and may select the externalapparatus 2000 for performing the checked operation.

In operation S416, the device 1000 generates control information aboutthe selected external apparatus 2000. If the condition determined by thedevice 1000 is satisfied, the device 1000 may generate the controlinformation for operating the device 1000 or the external apparatus2000. The device 1000 may generate the control information for operatingthe device 1000 or the external apparatus 2000, based on the operationinformation included in the user intention information and the apparatusinformation received from the device 1000. The control information mayinclude control commands about at least one external apparatus 2000 andthe device 1000, and the control commands may be aligned in apredetermined order.

In operation S418, the device 1000 provides a predetermined controlcommand to the external apparatus 2000 based on the generated controlinformation.

In operation S420, another external apparatus (not shown) isadditionally connected with the device 1000 and, in operation S422, thedevice 1000 generates control information including a control commandabout the additionally connected external apparatus. The device 1000 maychange the control information for satisfying the user's intention,based on apparatus information of the additionally connected externalapparatus. As such, the device 1000 may control operation of the device1000 or the external apparatus 2000 based on the changed controlinformation.

FIG. 5 is a flowchart of a method of directly determining a user'sintention and controlling the external apparatus 2000 by the device 1000based on user input information, according to an exemplary embodiment.

In operation S500, the device 1000 checks the external apparatus 2000controllable by the device 1000.

In operation S502, the device 1000 determines a user's intention basedon user input information. The device 1000 may determine the user'sintention by analyzing the user input information. In order to analyzethe user input information, the device 1000 may use various naturallanguage analysis methods. For example, the device 1000 may analyze textdata as the user input information by using natural language processingsuch as morpheme analysis, syntax analysis, or named entity recognition.If the user input information is voice data, the device 1000 maytransform the voice data into text data, and may analyze the transformedtext data. The device 1000 may generate user intention information byanalyzing the user input information. For example, if the user inputinformation is “Wake me up at 7:00 if it doesn't rain tomorrow”, thedevice 1000 may generate the user intention information including timeinformation such as “tomorrow, 7:00”, condition information such as “ifit doesn't rain”, and operation information such as “wake me up”.

In operation S504, the device 1000 determines a condition forcontrolling the external apparatus 2000 based on the user's intentionand, in operation S506, the device 1000 obtains condition information.In operation S508, the device 1000 obtains apparatus information of theexternal apparatus 2000, in operation S510, the device 1000 selects theexternal apparatus 2000 related to the user's intention and, inoperation S512, the device 1000 generates control information about theselected external apparatus 2000. In operation S514, the device 1000provides a predetermined control command to the external apparatus 2000based on the generated control information.

Operations S504, S506, S508, S510, S512, and S514 may correspond to theoperations described above with reference to FIGS. 2, 3, and 4, andthus, are not described in detail again.

FIG. 6 is an operation table of the device 1000 for controlling theexternal apparatus 2000, according to an exemplary embodiment.Information about operation of the device 1000 in relation to user inputinformation is recorded in the operation table of the device 1000.

As illustrated in FIG. 6, the operation table 58 of the device 1000 mayinclude a user input information field 60, a user's intention field 62,an external apparatus controlling condition field 64, and an operationfield 66. The operation table of the device 1000 may be generated by atleast one of the device 1000 and the server 3000.

User input information obtained by the device 1000 is recorded in theuser input information field 60. If a user inputs text to the device1000, the text data input by the user may be recorded in the user inputinformation field 60. If the user input information is voice data, thevoice data may be transformed into text data, and the transformed textdata may be stored in the user input information field 60.

The user input information recorded in the user input information field60 may include, for example, “Wake me up at 7:00 if it doesn't raintomorrow”, “Let's go home if the traffic is not heavy before 20:00tomorrow”, and “Give me a video call if the baby wakes up”.

Information about a user's intention determined based on the user inputinformation may be recorded in the user's intention field 62. The userinput information may be analyzed by using natural language processingsuch as morpheme analysis, syntax analysis, or named entity recognition.For example, if the user input information is “Wake me up at 7:00 if itdoesn't rain tomorrow”, “When: tomorrow, 7:00, Condition: if it doesn'train, Do: wake me up” may be recorded in the user's intention field 62.

Information about a condition that has to be satisfied to control theexternal apparatus 2000 according to the user's intention may berecorded in the external apparatus controlling condition field 64.

For example, based on the information recorded in the user's intentionfield 62, the external apparatus controlling condition field 64 maycontain “Who: x, What: weather, When: tomorrow, Condition: sunny,cloudy, Where: device location”. That is, a subject condition such as“weather”, a time condition such as “tomorrow”, a weather condition suchas “sunny, cloudy”, and a place condition such as “device location” maybe recorded in the external apparatus controlling condition field 64. Inaddition to “Who”, “What”, “When”, “Condition”, and “Where”, variouscondition items may be further included in the external apparatuscontrolling condition field 64. If predetermined data is not recorded asa condition item, the server 3000 may request the device 1000 foradditional user input information about the condition item having norecorded data, and may receive the additional user input informationfrom the device 1000 to record predetermined data as the condition item.That is, if the user's intention is not determined based on the userinput information, or if a condition and operation for controlling theexternal apparatus 2000 according to the user's intention are notdetermined, the server 3000 may request the device 1000 for additionaluser input information.

The device 1000 may generate a control command for checking whether apredetermined condition is satisfied, based on the information recordedin the condition field 64.

Operation information of the device 1000 and the external apparatus 2000for determining whether the condition for controlling the externalapparatus 2000 is satisfied, and for operating the device 1000 and theexternal apparatus 2000 according to the user's intention may berecorded in the operation field 66.

For example, based on the information recorded in the user's intentionfield 62, “check time”, “check device location”, and “check weather” maybe recorded in the operation field 66. If the condition for controllingthe external apparatus 2000 is satisfied, as an operation of the device1000 and the external apparatus 2000, “output alarm information” may berecorded in the operation field 66.

Based on the operation information recorded in the operation field 66,control information of the device 1000 and the external apparatus 2000for performing the operation recorded in the operation field 66 may begenerated. The control information may include a plurality of controlcommands for controlling operation of at least one of the device 1000and the external apparatus 2000, and the plurality of control commandsmay be executed by at least one of the device 1000 and the externalapparatus 2000 in a predetermined order. For example, based on “checktime”, “check device location” and “check weather” recorded in theoperation field 66, a control command may be generated in such a waythat the device 1000 checks the time, the location of the device 1000,and the weather by using at least one of the device 1000, a separateserver (not shown), and the external apparatus 2000. For example, basedon “output alarm information” recorded in the operation field 66, acontrol command may be generated in such a way that the device 1000 andthe external apparatus 2000 output predetermined alarm information.

Although the device 1000 determines the user's intention and determinesthe condition and operation for controlling the external apparatus 2000in the above description, exemplary embodiments are not limited thereto.The determining of the condition and operation for controlling theexternal apparatus 2000 based on the user input information by thedevice 1000 may be included in the determining of the user's intention.

FIG. 7 is a schematic diagram showing that the external apparatus 2000is controlled according to user input information that is input to thedevice 1000, in a system for controlling the external apparatus 2000,according to an exemplary embodiment.

Referring to FIG. 7, if a user inputs voice data such as “Wake me up at7:00 if it doesn't rain tomorrow” to the device 1000, the device 1000may provide the voice data or text data transformed from the voice data,as user input information to the server 3000. The device 1000 mayprovide apparatus information of a humidity sensor 72, a lighting device74, and a pillow vibrator 76 connectable with the device 1000, to theserver 3000.

The server 3000 may determine the user's intention based on the userinput information, and may determine a condition for controlling theexternal apparatus 2000. The server 3000 may determine whether thedetermined condition is satisfied, may generate control information forcontrolling the device 1000 and the external apparatus 2000, and mayprovide the generated control information to the device 1000.

The device 1000 may determine whether the condition is satisfied, byobtaining weather information from a weather server 70 based on thereceived control information, and obtaining humidity information fromthe humidity sensor 72. If the condition is satisfied, the device 1000may transmit a control command instructing to power on to the lightingdevice 74 and/or may transmit a control command instructing to generatevibration to the pillow vibrator 76.

FIG. 8 is a schematic diagram showing that the external apparatus 2000is controlled according to user input information that is input to thedevice 1000, in a system for controlling the external apparatus 2000,according to an exemplary embodiment.

Referring to FIG. 8, if a user inputs voice data such as “Let's go homeif the traffic is not heavy before 21:00 tomorrow” to the device 1000,the device 1000 may provide the voice data or text data transformed fromthe voice data, as user input information to the server 3000. The device1000 may provide apparatus information of a motor vehicle 82 and anavigator 84 connectable with the device 1000, to the server 3000.

The server 3000 may determine the user's intention based on the userinput information, and may determine a condition for controlling theexternal apparatus 2000. The server 3000 may determine whether thedetermined condition is satisfied, may generate control information forcontrolling the device 1000 and the external apparatus 2000, and mayprovide the generated control information to the device 1000.

The device 1000 may determine whether the condition is satisfied, byobtaining traffic information about a route from a current location ofthe device 1000 to home, from a traffic information providing server 80based on the received control information. If the condition issatisfied, the device 1000 may transmit a control command instructing tostart an engine to the motor vehicle 82 and may transmit a controlcommand instructing to output route guidance information to thenavigator 84.

FIG. 9 is a schematic diagram showing that the external apparatus 2000is controlled according to user input information that is input to thedevice 1000, in a system for controlling the external apparatus 2000,according to an exemplary embodiment.

Referring to FIG. 9, if a user inputs voice data such as “Give me avideo call if the baby wakes up” to the device 1000, the device 1000 mayprovide the voice data or text data transformed from the voice data, asuser input information to the server 3000. The device 1000 may provideapparatus information of a photographing apparatus 90, a recordingapparatus 92, and a telephone 94 connectable with the device 1000, tothe server 3000.

The server 3000 may determine the user's intention based on the userinput information, and may determine a condition for controlling theexternal apparatus 2000. The server 3000 may determine whether thedetermined condition is satisfied, may generate control information forcontrolling the device 1000 and the external apparatus 2000, and mayprovide the generated control information to the device 1000.

The device 1000 may determine whether the condition is satisfied, byobtaining a captured image from the photographing apparatus 90 andobtaining recorded voice data from the recording apparatus 92 based onthe received control information. If the condition is satisfied, thedevice 1000 may transmit a control command instructing to make a videocall to the telephone 94.

FIG. 10 is a schematic diagram showing controlling of the externalapparatus 2000 based on a voice command and biometric information of auser, which is performed by the server 3000, according to one or moreexemplary embodiments.

Referring to FIG. 10, while the user of the device 1000 is jogging, theuser may input a voice command to the device 1000. Then, the device 1000may receive the voice command input by the user and obtain biometricinformation of the user from a wearable device 4000. In this case, thewearable device 4000 may be a device separate from the device 1000 or adevice included in the device 1000. Additionally, the device 1000 maytransmit the voice command and the biometric information to the server3000.

The server 3000 may receive the voice command and the biometricinformation from the device 1000, and receive condition information fromthe external apparatus 2000. In this case, the external apparatus 2000may be, for example, an apparatus located in a house of a user.

Then, the server 3000 may generate a control command for controlling theexternal apparatus 2000 in consideration of the voice command of theuser, the biometric information of the user, and a condition of theexternal apparatus 2000, and transmit the generated control command tothe external apparatus 2000. Thus, the device 1000 may control theexternal apparatus 2000 according to the voice command of the user inconsideration of a biometric state of the user.

For example, the server 3000 may pre-store reference values of pieces ofbiometric information for determining a state of the user. If the userinputs a voice command to the device 1000, the wearable device 4000 maysense biometric information of the user and provide the biometricinformation to the server 3000. Additionally, the server 3000 maydetermine a state of the user by comparing a piece of biometricinformation of the user to a reference value and analyzing whether adifference between the piece of biometric information and the referencevalue is equal to or greater than a certain value within a certain range(for example, whether a heart rate is equal to or greater than 30 bpm,whether a difference between a sensible temperature and a currenttemperature is greater than ±5° C., or the like). The server 3000 maydetermine an intention of the user in consideration of the voice commandof the user and the state of the user. The server 3000 may select theexternal apparatus 2000 which is a control target so as to satisfy theintention of the user, and determine operation of the selected externalapparatus 2000. Additionally, the server 3000 may generate controlinformation for controlling the external apparatus 2000 and transmit thegenerated control information to the external apparatus 2000.

If information for selecting the external apparatus 2000 is included inthe voice command of the user (for example, if the user says “turn on anair conditioner”), the server 3000 may select the external apparatus2000, specified by the voice command, as a control target. The server3000 may determine the state of the user as well as the biometricinformation of the user, in consideration of information received fromthe device 1000 or the external apparatus 2000. For example, the server3000 may determine that the state of the user is “hot”, based oninformation regarding a heart rate of the user, calories consumed by theuser, and a distance the user has jogged. Additionally, the server 3000may select an air conditioner as a control target, generate controlinformation for setting a set temperature of the air conditioner as 18°C. and transmit the control information to the air conditioner.

Additionally, for example, if the user inputs a voice command saying“I'll be back home within 30 minutes” to the device 1000, the wearabledevice 4000 may transmit biometric information of the user to the server3000. The server 3000 may determine that a sensible temperature of theuser is 35° C. (which is a state of the user) by using the receivedbiometric information. Additionally, the server 3000 may select an airconditioner at home as a control target, and generate controlinformation for setting a set temperature of the air conditioner as 18°C.

If the user feels cold when the user is back home and changes the settemperature of the air conditioner to 22° C., the air conditioner maytransmit the temperature changed by the user to the server 3000. Theserver 3000 may change a setting for operation of the air conditioneraccording to a condition for controlling the air conditioner, by usinginformation about the temperature changed by the user. The server 3000may change a set value for ‘setting a set temperature of the airconditioner to 18° C. when a sensible temperature is 35° C.’ to a setvalue for ‘changing a set temperature of the air conditioner to 22° C.when a sensible temperature is 35° C.’. Then, if a sensible temperatureof the user is 35° C. when the user inputs a voice command saying “I'llbe back home within 30 minutes”, the server 3000 may generate a controlcommand instructing the air conditioner to set a temperature to 22° C.,instead of a control command instructing the air conditioner to set atemperature to 18° C.

FIG. 11 is a detailed flowchart of a method of controlling the externalapparatus 2000 according to a voice command of a user in considerationof biometric information of the user, which is performed by the server3000, according to one or more exemplary embodiments.

In operation S1100, the device 1000 may provide apparatus information ofthe external apparatus 2000 and device information of the wearabledevice 4000 to the server 3000. The device 1000 provides the apparatusinformation of the external apparatus 2000, located inside or outside ahouse of the user, to the server 3000. The external apparatus 2000 mayinclude, for example, a home appliance such as a smart TV or an airconditioner, a security camera, a recorder, a mobile phone, a PC, apillow vibrator, a motor vehicle, a navigator, a microphone, a speaker,a pedal, a joystick, a musical instrument (e.g., a piano, an organ, anelectronic keyboard, a guitar, a violin, or a cello), a gamemanipulator, a doll, a medical appliance, sporting equipment, a camera,or a sensor, but is not limited thereto. The apparatus information ofthe external apparatus 2000 may include information about at least oneselected from the group consisting of an identification value of theexternal apparatus 2000, a MAC address, an SSID, a type of the externalapparatus 2000, a capability provided by the external apparatus 2000, acategory, and a command used to control the external apparatus 2000.

Additionally, the device 1000 may provide the device information of thewearable device 4000 worn by the user to the server 3000. The wearabledevice 4000 may include, for example, a smart watch, smart glasses, asmart band, a smart earphone, smart shoes, a smart ring, a smartbracelet, or a smart necklace. The device information of the wearabledevice 4000 may, for example, include information about at least oneselected from the group consisting of an identification value of thewearable device 4000, a MAC address, an SSID, a type of the wearabledevice 4000, a capability provided by the wearable device 4000, acategory, and a command used to control the wearable device 4000.

The apparatus information of the external apparatus 2000 and the deviceinformation of the wearable device 4000, provided to the server 3000 bythe device 1000, may be used for the server 3000 to set a condition forcontrolling the external apparatus 2000.

In operation S1105, the device 1000 may provide voice commandinformation to the server 3000. The device 1000 may record a voicecommand of the user, and transmit the recorded voice command to theserver 3000. However, the providing of the voice command information isnot limited thereto. The device 1000 may convert the recorded voicecommand into text and provide the text, obtained by the converting, tothe server 3000. For example, the device 1000 may provide informationindicating a voice command of the user such as “I'll be back home within30 minutes' or ‘turn on an air conditioner’ to the server 3000.

In operation S1105, the device 1000 is described as recording a voicecommand and transmitting the recorded voice command to the server 3000.However, embodiments are not limited thereto. The wearable device 4000may record a voice command of the user, and transmit the recorded voicecommand to the device 1000. Then, the device 1000 may transmit therecorded voice command to the server 3000. Alternatively, the wearabledevice 4000 may record a voice command and transmit the recorded voicecommand to the server 3000.

In operation 1110, the wearable device 4000 may obtain biometricinformation of the user. For example, if the wearable device 4000 is asmart band, the wearable device 4000 may sense a heart rate or a bodytemperature of the user. Additionally, for example, if the wearabledevice 4000 is a glove or a ring, the wearable device 4000 may sense ablood flow rate at an end of the user's hand. However, the biometricinformation is not limited thereto, and the wearable device 4000 mayobtain various biometric information regarding a brain wave or a motionof the user. Additionally, the wearable device 4000 may sense biometricinformation of the user for a predetermined period of time or when apredetermined event occurs. In this case, a predetermined period of timeor a predetermined event may be preset by the server 3000 or the device1000.

In operation S1115, the server 3000 may determine a condition forcontrolling the external apparatus 2000. The server 3000 may analyzeinformation about the voice command and determine a condition forcontrolling the external apparatus 2000 with respect to the voicecommand of the user, according to a type of biometric information thatmay be received from the wearable device 4000 and a type of the externalapparatus 2000. Additionally, the server 3000 may determine a conditionfor controlling the external apparatus 2000 based on conditioninformation collected from at least one selected from the groupconsisting of the device 1000 and the external apparatus 2000.

The server 3000 may determine an intention of the user, in considerationof the voice command and the biometric information of the user.Additionally, the server may determine a condition for controlling theexternal apparatus 2000, so as to satisfy the intention of the user.Additionally, the server 3000 may select the external apparatus 2000that is to be a control target. Additionally, if the condition forcontrolling the external apparatus 2000 is satisfied, the server 3000may determine operation of the external apparatus 2000 which is to becontrolled by the server 3000.

For example, if a voice command of the user is “I'll be back home within30 minutes′, the wearable device 4000 is a smart band, and one or moreexternal apparatuses 2000 in the house of the user is at least one of anair conditioner 110-2, a heater 110-3, and a temperature sensor 110-1,and if the server 3000 determines that a heart rate of the user isgreater than 120 bpm, that calories consumed by the user are greaterthan 500 kcal, that a moving distance of the user is greater than 3 km,because the user usually feels hot, the server 3000 may determine to seta target temperature of the air conditioner to 18° C.

Additionally, for example, if a voice command of the user is “turn onthe air conditioner', the wearable device 4000 is a glove or a ring, andone or more external apparatuses 2000 in the house of the user is atleast one of an air conditioner, a heater, and a temperature sensor, andif the server 3000 determines that a blood flow rate at an end of theuser's hand is greater than a reference value, that calories consumed bythe user are greater than 500 kcal, that a moving distance of the useris greater than 3 km, and that a sensible temperature of the user isgreater than 35° C., the server 3000 may determine to set a targettemperature of the air conditioner to 18° C.

In operation S1120, the wearable device 4000 may transmit the biometricinformation of the user to the device 1000. In operation S1125, thedevice 1000 may transmit the biometric information of the user to theserver 3000. The device 1000 may transmit condition information such asa number of steps taken by the user and calories consumed by the user tothe server 3000. Additionally, the wearable device 4000 may provide thebiometric information of the user for a predetermined period of time orwhen a predetermined event occurs. In this case, the predeterminedperiod of time or the predetermined event may be preset by the server3000 or the device 1000.

The device 1000 may receive the voice command, request the wearabledevice 4000 for the biometric information of the user, and receive thebiometric information which is obtained by the wearable device 4000 fromthe wearable device 4000. Additionally, the device 1000 maysimultaneously transmit the voice command and the biometric informationto the server 3000. Alternatively, the device 1000 may transmit thebiometric information to the server 3000 after the device 1000 transmitsthe voice command to the server 3000.

Alternatively, the wearable device 4000 may receive the voice command ofthe user, obtain the biometric information in response to receiving thevoice command, and transmit the voice command and the biometricinformation to the device 1000. Then, the device 1000 may transmit thevoice command and the biometric information to the server 3000.

If the wearable device 4000 is included in the device 1000, the device1000 may receive the voice command of the user, obtain the biometricinformation in response to receiving the voice command, and transmit thevoice command and the biometric information to the server 3000.

In operation S1130, a temperature sensor 110-1 may provide temperatureinformation to the server 3000. The temperature sensor 110-1 in thehouse may sense a temperature in the house, and provide informationabout the sensed temperature to the server 3000. The temperature sensor110-1 may provide the information about the sensed temperature for apredetermined period of time or when a predetermined event occurs. Inthis case, the predetermined period of time or the predetermined eventmay be preset by the server 3000 or the device 1000.

In operation S1135, the server 3000 may determine whether the conditionfor controlling the external apparatus 2000 is satisfied. The server3000 may determine whether the condition, determined in operation S1115,is satisfied based on the biometric information received from thewearable device 4000 and the condition information received from the atleast one selected from the group consisting of the external apparatus2000 and the device 1000. The server 3000 may determine whether thecondition for controlling the external apparatus 2000 is satisfied, inconsideration of a heart rate of the user, a blood flow rate at an endof the user's hand, a number of steps taken by the user, a sensibletemperature of the user, and a condition inside and outside the house.

Additionally, the server 3000 may determine whether the condition forcontrolling the external apparatus 2000 is satisfied, by analyzing thebiometric information received from the wearable device 4000 and thecondition information received from the at least one selected from thegroup consisting of the external apparatus 2000 and the device 1000.

In operation S1140, the server 3000 may generate control information forcontrolling the external apparatus 2000. The control information mayinclude control commands with respect to the external apparatus 2000 andthe device 1000. The control commands may be arranged according to acertain order. For example, if an operation of the external apparatus2000 to be controlled by the server 3000 is an ‘operation of setting atarget temperature of an air conditioner to 18° C.’, the server 3000 maygenerate control information for controlling the air conditioner, byarranging a control command for turning a power of the air conditioneron and a control command for setting a target temperature of the airconditioner to 18° C.

In operation S1145, the server 3000 may provide the control informationto the external apparatus 2000. For example, the server 3000 may providethe control information for setting the target temperature to 18° C. tothe air conditioner. The server 3000 may transmit the controlinformation directly to the external apparatus 2000. However, thetransmitting of the control information is not limited thereto. Theserver 3000 may provide the control information to the externalapparatus 2000 via the device 1000 or a gateway (not shown).

The server 3000 may update the condition for controlling the externalapparatus 2000 and a set value for operation of the external apparatus2000, based on feedback information of the user with respect to controlof the external apparatus 2000. For example, if the air conditioner iscontrolled to set a set temperature to 18° C. when a sensibletemperature of the user is 35° C. and the user changes the settemperature of the air conditioner to 22° C. within a predeterminedperiod of time after the user is back home, the air conditioner maytransmit the set temperature changed by the user to the server 3000.Thus, the server 3000 may change a set value for ‘setting a settemperature of the air conditioner to 18° C. when a sensible temperatureis 35° C.’ to a set value for ‘changing a set temperature of the airconditioner to 22° C. when a sensible temperature is 35° C.’. Then, if asensible temperature of the user is 35° C. when the user gives a commandsuch as “I'll be back home within 30 minutes”, the server 3000 maygenerate and transmit a control command for setting a temperature to 22°C. to the air conditioner, instead of transmitting a control command forsetting a temperature to 18° C. to the air conditioner.

FIG. 12 is a schematic diagram showing an example of an operation tableof the device 1000 for controlling the external apparatus 2000 based onbiometric information, according to one or more exemplary embodiments.

Referring to FIG. 12, according to one or more exemplary embodiments,the operation table may include a user voice command field 120, abiometric information obtainer field 121, a condition information field122, a biometric information field 123, an external apparatuscontrolling condition field 124, an operation field 125, and a controltarget apparatus field 126.

Information indicating a voice command of a user may be written to theuser voice command field 120. Information indicating a voice command ofthe user such as ‘I'll be back within 30 minutes’ or ‘turn on the airconditioner’ may be written to the user voice command field 120.

An identification value or a type of the wearable device 4000 or thedevice 1000 which may obtain biometric information of the user may bewritten to the biometric information obtainer field 121. For example, asmart band, a glove, or a ring may be written to the biometricinformation obtainer field 121.

Information indicating a type of condition information which may be usedto determine a condition for controlling the external apparatus 2000 ordetermine whether a condition for controlling the external apparatus2000 is satisfied may be written to the condition information field 122.For example, information such as a temperature in a house or a locationof the device 1000 may be written to the condition information field122.

A type of biometric information that may be obtained from the wearabledevice 4000 may be written to the biometric information field 123. Aheart rate, a blood flow rate at an end of a hand, and calorieconsumption may be written to the biometric information field 123.

Information indicating a condition for controlling the externalapparatus 2000 may be written to the external apparatus controllingcondition field 124. For example, a condition such as ‘a heart rate:greater than 120 bpm, calorie consumption: 500 kcal, and a movingdistance: greater than 3 km’ may be written to the external apparatuscontrolling condition field 124.

Information about operation of the external apparatus 2000 which is tobe controlled when the condition for controlling the external apparatus2000 is satisfied may be written to the operation field 125. Informationsuch as ‘air conditioner power on, temperature set to 18° C.’ may bewritten to the operation field 125.

An identification value of the external apparatus 2000 that is to becontrolled by the server 3000 may be written to the control targetapparatus field 126.

FIG. 13 is a schematic diagram showing an example of controlling theexternal apparatus 2000 based on a voice command and biometricinformation of a user, which is performed by the server 3000, accordingto one or more exemplary embodiments.

Referring to FIG. 13, the device 1000 may receive a user input of avoice command and a user input of setting a health state of the user.Additionally, the device 1000 may transmit a voice command and healthinformation to the server 3000. The health information of the user maybe information regarding a health state of the user, and includeinformation about a name of a disease, a taste, an age, a height, or aweight of the user. The health information of the user may be used todetermine a health state of the user.

The server 3000 may receive a voice command and health information fromthe device 1000, and receive condition information from the externalapparatus 2000. In this case, the external apparatus 2000 may be, forexample, an apparatus located in a house of the user. Additionally, theserver 3000 may receive health information of the user from a hospitalserver (not shown). In this case, the server 3000 may obtain anauthority for accessing the health information of the user which isstored in the hospital server (not shown).

Then, the server 3000 may generate a control command for controlling theexternal apparatus 2000 in consideration of the voice command of theuser, the health information of the user, and a condition of theexternal apparatus 2000, and transmit the generated control command tothe external apparatus 2000.

Thus, the server 3000 may control the external apparatus 2000 accordingto a voice command of the user, in consideration of the health state ofthe user.

For example, if the user inputs health information indicating ‘asthma’to the device 1000 and inputs a voice command saying “I'll be back homewithin 30 minutes”, the server 3000 may receive health information andvoice command information from the device 1000, and thus, analyze anintention of the user. Additionally, the server 3000 may generate acontrol command for selecting an air purifier 140-4 (see FIG. 14) fromamong one or more external apparatuses 2000 located in the house andoperating the air purifier 20 minutes before the user is back home.Additionally, the server 3000 may transmit the generated control commandto the air purifier. In this case, the user may additionally inputanother piece of health information to the device 1000, and a pluralityof pieces of health information may be used by the server 3000.

FIG. 14 is a detailed flowchart of a method of controlling the externalapparatus 2000 according to a voice command of a user in considerationof health information about the user, which is performed by the server3000, according to one or more exemplary embodiments.

In operation S1400, the device 1000 may provide apparatus information ofthe external apparatus 2000 to the server 3000. The device 1000 mayprovide the apparatus information of the external apparatus 2000,located inside or outside a house of the user, to the server 3000. Theapparatus information of the external apparatus 2000, provided to theserver 3000 by the device 1000, may be used to set a condition forcontrolling the external apparatus 2000.

In operation S1405, the device 1000 may provide health information ofthe user to the server 3000. The device 1000 may display a graphicaluser interface (GUI) for receiving an input of the health information ona screen of the device 1000, and the user may input a health state ofthe user to the device 1000 via the displayed GUI.

In operation S1410, the device 1000 may provide voice commandinformation to the server 3000. The device 1000 may record a voicecommand of the user, and transmit the recorded voice command to theserver 3000. However, the providing of the voice command information isnot limited thereto. The device 1000 may convert the recorded voicecommand into text and provide the text, obtained by the converting, tothe server 3000. For example, the device 1000 may provide informationindicating a voice command of the user such as “I'll be back home within30 minutes' to the server 3000.

In operation S1410, the device 1000 is described as recording a voicecommand and transmitting the recorded voice command to the server 3000.However, embodiments are not limited thereto. The wearable device 4000may record a voice command of the user, and transmit the recorded voicecommand to the device 1000. Then, the device 1000 may transmit therecorded voice command to the server 3000. Alternatively, the wearabledevice 4000 may record a voice command and transmit the recorded voicecommand to the server 3000.

In operation S1415, the server 3000 may determine a condition forcontrolling the external apparatus 2000. The server 3000 may analyzeinformation about the voice command and determine a condition forcontrolling the external apparatus 2000 with respect to the voicecommand of the user, according to a type of the external apparatus 2000and a health state of the user. Additionally, the server 3000 maydetermine a condition for controlling the external apparatus 2000 basedon condition information collected from at least one selected from thegroup consisting of the device 1000 and the external apparatuses 2000.

The server 3000 may determine an intention of the user, in considerationof the voice command and the biometric information of the user.Additionally, the server 3000 may determine a condition for controllingthe external apparatus 2000, so as to satisfy the intention of the user.

Additionally, the server 3000 may select the external apparatus 2000that is to be a control target, in consideration of a condition insideor outside the house. Additionally, if the condition for controlling theexternal apparatus 2000 is satisfied, the server 3000 may determineoperation of the external apparatus 2000 that is to be controlled by theserver 3000. For example, the server 3000 may determine the externalapparatus 2000 that is to be a control target and operation of theexternal apparatus 2000. In this case, an optimum environment value inthe house according to a season and a temperature outside the house maybe preset.

For example, if a voice command of the user is ‘I'll be back home within30 minutes’, a health state of the user is ‘asthma’, and one or more ofthe external apparatuses 2000 in the house of the user include an airconditioner (see FIG. 11), a heater, an air purifier, a temperaturesensor, or a humidity sensor, and if the server 3000 determines that theuser is near the house, the server 3000 may determine to turn on thepower of the air purifier.

Additionally, for example, if a voice command of the user is ‘I'll beback home within 30 minutes’, a health state of the user is ‘atopy, acurrent season is ‘summer’, and one or more of the external apparatuses2000 at the user's home include at least one of an air conditioner, aheater, an air purifier, a humidifier, a dehumidifier, a temperaturesensor, and a humidity sensor, and if the server 3000 determines that atemperature in the house is greater than 30° C. and humidity in thehouse is less than 50%, the server 3000 may determine to set atemperature of the air conditioner to 27° C., and a humidity level of adehumidifier or a humidifier 140-5 to 55%.

Additionally, for example, if a voice command of the user is ‘I'll beback home within 30 minutes’, a health state of the user is ‘allergicrhinitis’, a current season is winter, and one or more of the externalapparatuses 2000 at the user's home include at least one of an airconditioner, a heater, an air purifier, a dehumidifier, a temperaturesensor, and a humidity sensor, and if the server 3000 determines that atemperature in the house is greater than 15° C. and humidity in thehouse is less than 30%, the server 3000 may determine to set atemperature of the heater 140-3 to 20° C., and a humidity level of adehumidifier to 40%.

Additionally, the server 3000 may store set values for an optimum indoortemperature and optimum indoor humidity for a general adult according toseasons. Since a heat retention rate of clothes worn by people variesdepending on seasons and temperature and humidity are associated with aperson's comfort, optimum humidity may vary depending on a change in anoptimum temperature. For example, in spring and fall, an optimumtemperature may range from 19° C. to 23° C., and optimum humidity may be50%. In summer, an optimum temperature may range from 24° C. to 27° C.,and optimum humidity may be 60%. In winter, an optimum temperature mayrange from 18° C. to 21° C., and optimum humidity may be 40%.Additionally, the set values for an optimum indoor temperature and anoptimum indoor humidity for a general adult according to seasons may beused for the server 3000 to determine a condition for controlling theexternal apparatus 2000 and operation of the external apparatus 2000.

Additionally, an optimum temperature and an optimum humidity for theuser may be different from an optimum temperature and an optimumhumidity for general adults, according to characteristics of the user(for example, the user's age, gender, height, weight, chronic disease,and condition). For example, if the season is summer and a disease ofthe user is atopy, an optimum indoor temperature for the user may be setto 20° C., and optimum humidity may be set to 55%.

For example, if a user who is an atopic patient inputs a voice commandsaying “I'll be back home within 30 minutes” in summer, the server 3000may analyze an intention of the user in consideration of the voicecommand saying “I'll be back home within 30 minutes”. Additionally, theserver 3000 may select an air conditioner and a dehumidifier from amongone or more external apparatuses 2000, in consideration of the intentionof the user, ‘summer’, and ‘atopy’, and generate control information soas to set a temperature of the air conditioner to 20° C. and a humiditylevel of the dehumidifier to 55%, 20 minutes before the user arrives athome.

Additionally, for example, if a season is ‘winter’ and a health state ofthe user is ‘allergic rhinitis’, an optimum indoor humidity for the usermay be set to 50% so that the optimum indoor humidity for the user ishigher than optimum humidity for general adults. Accordingly, if a userwho suffers from allergic rhinitis inputs a voice command saying “I'llbe back home within 30 minutes” in winter, the server 3000 may analyzean intention of the user in consideration of the voice command such that“I'll be back home within 30 minutes”. Additionally, the server 3000 mayselect a heater and a dehumidifier from among one or more externalapparatuses 2000 in a house of the user, in consideration of theintention of the user, ‘winter’, and ‘atopy’, and generate controlinformation so as to set a temperature of the air conditioner to 20° C.and a humidity level of the dehumidifier to 55%, 20 minutes before theuser arrives home.

The server 3000 may use various sensing values received from theexternal apparatus 2000 to determine a condition for controlling theexternal apparatus 2000 and operation of the external apparatus 2000.Additionally, various set values for the condition for controlling theexternal apparatus 2000 and operation of the external apparatus 2000 maybe stored according to a season, a health state of the user, biometricinformation of the user, or a condition inside or outside the house.

In operation S1420, a temperature sensor 140-1 in the house may providetemperature information to the server 3000. The temperature sensor 140-1in the house may sense a temperature in the house, and providetemperature information, obtained by the sensing, to the server 3000.The temperature sensor 140-1 may provide the temperature information fora predetermined period of time or when a predetermined event occurs. Inthis case, the predetermined period of time or the predetermined eventmay be preset by the server 3000 or the device 1000.

In operation S1425, a humidity sensor 140-2 in the house may providehumidity information to the server 3000. The temperature sensor 140-1 inthe house may sense a temperature in the house, and provide temperatureinformation, obtained by the sensing, to the server 3000. The humiditysensor 140-2 may provide humidity information for a predetermined periodof time or when a predetermined event occurs. In this case, thepredetermined period of time or the predetermined event may be preset bythe server 3000 or the device 1000.

In operation S1430, the server 3000 may determine whether the conditionfor controlling the external apparatus 2000 is satisfied. The server3000 may determine whether the condition, determined in operation S1415,is satisfied based on a health state of the user and the conditioninformation collected from at least one selected from the groupconsisting of the device 1000 and the external apparatuses 2000.

In operation S1435, the server 3000 may generate control information forcontrolling the external apparatus 2000. The control information mayinclude control commands with respect to the external apparatus 2000 andthe device 1000. The control commands may be arranged according to acertain order. For example, if operation of the external apparatus 2000to be controlled by the server 3000 is ‘operation of setting atemperature of an air conditioner to 27° C. and setting a humidity levelof the dehumidifier to 55%’, the server 3000 may generate controlinformation for controlling the air conditioner and the dehumidifier, byarranging a control command for turning on a power of the airconditioner, a control command for setting a target temperature of theair conditioner to 27° C., a control command for turning on thedehumidifier, and a control command for setting a target humidity levelof the dehumidifier to 55%.

In operations S1440, S1445, and S1450, the server 3000 may provide thecontrol information to the external apparatus 2000. The server maytransmit the control information directly to the external apparatus2000. However, the transmitting of the control information is notlimited thereto. The server 3000 may provide the control information tothe external apparatus 2000 via the device 1000 or a gateway (notshown).

FIG. 15 is a schematic diagram showing an example of an operation tableof the device 1000 for controlling the external apparatus 2000 based ona health state of a user, according to one or more exemplaryembodiments.

Referring to FIG. 15, according to one or more exemplary embodiments,the operation table may include a user voice command field 150, a userhealth state field 151, a condition information field 152, an externalapparatus controlling condition field 153, an operation field 154, and acontrol target apparatus field 155.

Information indicating a voice command of a user may be written to theuser voice command field 150. Information indicating a voice command ofthe user saying ‘I'll be back within 30 minutes’ or ‘turn on the airconditioner’ may be written to the user voice command field 150.

Information indicating a health state of the user may be written to theuser health state field 151. The device 1000 may display a GUI forsetting a health state of the user on a screen of the device 1000, andset a health state of the user based on a user input via the GUI.Additionally, information about the set health state may written to theuser health state field 151. Information such as ‘asthma’, ‘atopy’, or‘allergic rhinitis’ may be written to the user health state field 151.

Information indicating a type of condition information which may be usedto determine a condition for controlling the external apparatus 2000 ordetermine whether a condition for controlling the external apparatus2000 is satisfied may be written to the condition information field 152.For example, information such as a temperature in a house, humidity inthe house, or a location of the device 1000 may be written to thecondition information field 152.

Information indicating a condition for controlling the externalapparatus 2000 may be written to the external apparatus controllingcondition field 153. A condition such as ‘a location of the user: nearhome’, ‘a temperature in the house: greater than 30° C.’, or ‘humidityin the house: less than 50%’ may be written to the external apparatuscontrolling condition field 153.

Information about operation of the external apparatus 2000 which is tobe controlled when the condition for controlling the external apparatus2000 is satisfied may be written to the operation field 154. Informationsuch as ‘air purifier power on, air conditioner power on, temperatureset to 27° C., dehumidifier power on, or humidity level set to 55%’ maybe written to the operation field 125.

An identification value of the external apparatus 2000 that is to becontrolled by the server 3000 may be written to the control targetapparatus field 155.

FIGS. 16 and 17 are schematic diagrams showing an example of receiving avoice command of a user, which is performed by the device 1000,according to one or more embodiments.

Referring to FIG. 16, the user may execute a voice recognitionapplication installed on the device 1000, and touch a button 160 forinputting a voice command on an execution screen of the voicerecognition application.

Referring to FIG. 17, as the button 160 for inputting a voice command istouched, the device 1000 may display a GUI for guiding a voice input ofthe user on a screen of the device 1000. Then, the user may input avoice command to the device 1000 via the displayed GUI.

FIGS. 18, 19, and 20 are schematic diagrams showing an example ofreceiving a user input of health information of a user, which isperformed by the device 1000, according to one or more exemplaryembodiments.

Referring to FIGS. 18 through 20, the user may execute a voicerecognition application installed on the device 1000, and sequentiallytouch buttons 180, 190 and 200 for inputting health information of theuser on an execution screen of the voice recognition application.

Referring to FIG. 21, as the buttons 180 through 200 for inputtinghealth information are sequentially touched, the device 1000 may displaya selection menu for selecting a health state of the user on a screen ofthe device 1000. The selection menu for selecting a health state of theuser may include items such as asthma, a baby, atopy 210, allergicrhinitis, asthma, or a vegetarian. For example, the user may select theatopy 210 from the selection menu.

Referring to FIG. 22, as the atopy 210 is selected from the selectionmenu for selecting a health state of the user, the device 1000 mayselect a health state of the user as ‘atopy’.

FIG. 23 is a block diagram of the device 1000 according to an exemplaryembodiment.

As illustrated in FIG. 23, the device 1000 includes an externalapparatus checker 1100, an input unit 1200, an information provider1300, a condition information obtainer 1400, a control informationobtainer 1500, a control command provider 1600, a database (DB) 1700, atransceiver 1800, and a controller 1900.

The external apparatus checker 1100 checks the external apparatus 2000controllable by the device 1000. The external apparatus checker 1100 maycheck the external apparatus 2000 controllable by the device 1000 fromamong the external apparatuses connectable with the device 1000. Forexample, the external apparatus checker 1100 may identify or select theexternal apparatus 2000 based on criteria, such as connectivity,compatibility, ability to share functionality, etc., with the device1000.

The external apparatus checker 1100 may receive from the checkedexternal apparatus 2000 apparatus information of the external apparatus2000, but is not limited thereto. The external apparatus checker 1100may receive the apparatus information of the external apparatus 2000from a separate server (not shown). For example, if the device 1000 isconnected with the external apparatus 2000 via a home gateway (notshown), the external apparatus checker 1100 may request the home gatewayfor the apparatus information of the external apparatus 2000. Theapparatus information of the external apparatus 2000 may includeinformation about at least one of an identification value of theexternal apparatus 2000, the type of the external apparatus 2000, and acommand used to control the external apparatus 2000.

The input unit 1200 receives a user input about the device 1000, andgenerates user input information. The user input information may includetext data or voice data, but is not limited thereto. Alternatively, theinput unit 1200 may generate the user input information by using anemail or a text message stored in the device 1000.

The information provider 1300 provides to the server 3000 various typesof information for generating control information. The informationprovider 1300 may provide to the server 3000 at least one of theapparatus information of the external apparatus 2000, the user inputinformation, and condition information.

The condition information obtainer 1400 may obtain the conditioninformation from at least one of the device 1000, a separate server (notshown), and the external apparatus 2000. The condition informationobtainer 1400 may obtain the condition information based on the controlinformation obtained by the control information obtainer 1500 to bedescribed below, but is not limited thereto. In order to satisfy auser's intention, the condition information obtainer 1400 may obtain thecondition information related to a condition for controlling theexternal apparatus 2000.

The control information obtainer 1500 obtains the control informationfor controlling the device 1000 and the external apparatus 2000according to the user's intention. The control information obtainer 1500may receive the control information generated by the server 3000, but isnot limited thereto. The control information obtainer 1500 may determinethe user's intention based on the user input information, may determinethe condition for controlling the external apparatus 2000, and maygenerate the control information. The control information obtainer 1500may receive information about at least one of the condition forcontrolling the external apparatus 2000 and the user's intention fromthe server 3000, and may generate the control information. The controlinformation obtainer 1500 will be described in detail below withreference to FIG. 24.

The control command provider 1600 provides a predetermined controlcommand to the external apparatus 2000 based on the obtained controlinformation. The control information may include a plurality of controlcommands aligned in a predetermined order, and the control commandprovider 1600 may provide the plurality of control commands to at leastone external apparatus 2000 in a predetermined order.

The DB 1700 stores various types of information to control the device1000 and the external apparatus 2000 by the device 1000 according to theuser's intention.

The transceiver 1800 transmits and receives to and from the externalapparatus 2000 and the server 3000 the various types of informationneeded to control the device 1000 and the external apparatus 2000 by thedevice 1000 according to the user's intention.

The controller 1900 controls operations of the device 1000, and controlsthe external apparatus checker 1100, the input unit 1200, theinformation provider 1300, the condition information obtainer 1400, thecontrol information obtainer 1500, the control command provider 1600,the DB 1700, and the transceiver 1800 in such a way that the device 1000controls the device 1000 and the external apparatus 2000 according tothe user's intention.

Some or all of the external apparatus checker 1100, the input unit 1200,the information provider 1300, the condition information obtainer 1400,the control information obtainer 1500, and the control command provider1600 may be driven by software modules, but this is not limiting. Someor all of the external apparatus checker 1100, the input unit 1200, theinformation provider 1300, the condition information obtainer 1400, thecontrol information obtainer 1500, and the control command provider 1600may be hardware devices.

Also, at least some of the external apparatus checker 1100, the inputunit 1200, the information provider 1300, the condition informationobtainer 1400, the control information obtainer 1500, and the controlcommand provider 1600 may be included in the controller 1900, and theexternal apparatus checker 1100, the input unit 1200, the informationprovider 1300, the condition information obtainer 1400, the controlinformation obtainer 1500, the control command provider 1600, and thecontroller 1900 may be driven by one processor. However, an exemplaryembodiment is not limited thereto.

FIG. 24 is a block diagram of the condition information obtainer 1400according to an exemplary embodiment.

As illustrated in FIG. 24, the condition information obtainer 1400 mayinclude a user intention determiner 1410, an apparatus control conditiondeterminer 1420, a control information generator 1430, and a controlinformation receiver 1440.

The user intention determiner 1410 determines a user's intention byanalyzing the user input information. For example, the user intentiondeterminer 1410 may use various natural language analysis methods. Forexample, the device 1000 may analyze text data as the user inputinformation by using natural language processing such as morphemeanalysis, syntax analysis, or named entity recognition. If the userinput information is voice data, the user intention determiner 1410 maytransform the voice data into text data, and may analyze the transformedtext data. The user intention determiner 1410 may generate userintention information by analyzing the user input information. Forexample, if the user input information is “Wake me up at 7:00 if itdoesn't rain tomorrow”, the user intention determiner 1410 may generatethe user intention information including time information such as“tomorrow, 7:00”, condition information such as “if it doesn't rain”,and operation information such as “wake me up”. Alternatively, the userintention determiner 1410 may receive from the server 3000 the userintention information generated by the server 3000.

The apparatus control condition determiner 1420 determines a conditionfor controlling the external apparatus 2000 based on the user'sintention. The apparatus control condition determiner 1420 may determinea condition that has to be satisfied to control operation of theexternal apparatus 2000 or the device 1000, based on the user intentioninformation. For example, if the user intention information includestime information such as “tomorrow, 7:00” and condition information suchas “if it doesn't rain”, the apparatus control condition determiner 1420may determine a time condition such as “tomorrow”, a weather conditionsuch as “sunny, cloudy”, and a place condition such as “device location”as the condition for controlling the external apparatus 2000.Alternatively, the apparatus control condition determiner 1420 mayreceive from the server 3000 information about the condition determinedthe server 3000.

The control information generator 1430 generates control informationabout the device 1000 and the external apparatus 2000. The controlinformation generator 1430 may determine whether the condition issatisfied, based on the condition information. If the conditiondetermined by the device 1000 is satisfied, the control informationgenerator 1430 may generate the control information for operating thedevice 1000 or the external apparatus 2000. The control informationgenerator 1430 may generate the control information for operating thedevice 1000 or the external apparatus 2000, based on operationinformation, and apparatus information of the external apparatus 2000,which are included in the user intention information. The controlinformation may include control commands about at least one of theexternal apparatus 2000 and the device 1000, and the control commandsmay be aligned in a predetermined order.

The control information receiver 1440 may receive from the server 3000the control information generated by the server 3000.

FIG. 25 is a block diagram of the device 1000 according to an exemplaryembodiment.

A mobile communicator 1001 performs, for example, call setup and datacommunication with a base station via a cellular network such as a 3G/4Gnetwork. A sub communicator 1002 performs a function for short-distancecommunication such as Bluetooth or NFC. A broadcasting receiver 1003receives a digital multimedia broadcasting (DMB) signal.

A camera 1004 includes a lens and optical elements for capturing a photoor video.

A sensor 1005 may include a gravity sensor for sensing motion of thedevice 1000, an illumination sensor for sensing brightness of light, aproximity sensor for sensing proximity of a person, a motion sensor forsensing motion of a person, etc.

A GPS receiver 1006 receives a GPS signal from a satellite. Variousservices may be provided to a user by using GPS signals.

An input/output (I/O) unit 1010 provides an interface with the externalapparatus 2000 or the user, and includes buttons 1011, a microphone1012, a speaker 1013, a vibration motor 1014, a connector 1015, and/or akeypad 1016.

A touch screen 1018 receives a touch input of the user. A touch screencontroller 1017 transmits to a controller 1050 the touch input receivedby the touch screen 1018. A power supply source 1019 is connected with abattery or an external power source to supply power to the device 1000.

The controller 1050 provides a control command to the external apparatus2000 by executing programs stored in a memory 1060.

The programs stored in the memory 1060 may be classified into aplurality of modules according to their functions, for example, a mobilecommunication module 1061, a Wi-Fi module 1062, a Bluetooth module 1063,a DMB module 1064, a camera module 1065, a sensor module 1066, a GPSmodule 1067, a video player module 1068, an audio player module 1069, apower module 1070, a touch screen module 1071, a user interface (UI)module 1072, and an application module 1073.

Functions of most of the modules would be instinctively understood byone of ordinary skill in the art in view of their names and thus onlythe application module 1073 will be described here. The applicationmodule 1073 may check the external apparatus 2000 connected with thedevice 1000 by using the mobile communicator 1001 and the subcommunicator 1002, and may receive apparatus information from theexternal apparatus 2000. The application module 1073 may receive a userinput by using the I/O unit 1010, and may generate user inputinformation. Furthermore, the application module 1073 may directlyobtain condition information via the camera 1004, the sensor 1005, orthe GPS receiver 1006, or may receive the condition information from theexternal apparatus 2000 or a separate server (not shown) by using themobile communicator 1001 and the sub communicator 1002. Besides, theapplication module 1073 may obtain control information for controllingthe device 1000 and the external apparatus 2000. The application module1073 may determine the user's intention based on the user inputinformation or may receive user intention information from the server3000. The application module 1073 may determine a condition forcontrolling the external apparatus 2000 based on the user's intention,or may receive information about the condition from the server 3000. Theapplication module 1073 may determine whether the condition is satisfiedand may generate the control information for controlling the device 1000and the external apparatus 2000. The application module 1073 may receivethe control information generated by the server 3000. Furthermore, theapplication module 1073 may provide to the external apparatus 2000 acontrol command included in the control information.

FIG. 26 is a block diagram of the server 3000 according to an exemplaryembodiment.

As illustrated in FIG. 26, the server 3000 includes an input informationreceiver 3100, an apparatus information receiver 3200, a user intentiondeterminer 3300, a condition information receiver 3400, a controlinformation generator 3500, an information provider 3600, a DB 3700, atransceiver 3800, and a controller 3900.

The input information receiver 3100 receives user input information fromthe device 1000. The user input information may include at least one oftext data and voice data, but is not limited thereto.

The apparatus information receiver 3200 receives apparatus informationof the external apparatus 2000 from the device 1000. The apparatusinformation may include information including at least one of anidentification value of the external apparatus 2000, the type of theexternal apparatus 2000, and a command used to control the externalapparatus 2000.

The user intention determiner 3300 determines a user's intention byanalyzing the user input information. For example, the user intentiondeterminer 3300 may use various natural language analysis methods.

The condition information receiver 3400 receives condition informationfrom the device 1000. The condition information receiver 3400 mayrequest the device 1000 for the condition information and may receivefrom the device 1000 the condition information obtained by the device1000.

The control information generator 3500 generates control information forcontrolling the device 1000 and the external apparatus 2000. The controlinformation generator 3500 may generate the control information based onat least one of the user intention information, the apparatusinformation, and the condition information. The control informationgenerator 3500 may determine the user's intention based on the userinput information, may determine the condition for controlling theexternal apparatus 2000, and may generate the control information.

The information provider 3600 may provide to the device 1000 at leastone of the user intention information, and the condition information andthe control information for controlling the external apparatus 2000.

The DB 3700 stores various types of information used when the server3000 generates the control information for operating the device 1000 andthe external apparatus 2000 according to the user's intention based onthe user input information, and provides the generated controlinformation to the device 1000.

The transceiver 3800 transmits to and receives from the device 1000 thevarious types of information used when the server 3000 generates thecontrol information for operating the device 1000 and the externalapparatus 2000 according to the user's intention based on the user inputinformation, and provides the generated control information to thedevice 1000.

The controller 3900 controls operations of the server 3000, and controlsthe input information receiver 3100, the apparatus information receiver3200, the user intention determiner 3300, the condition informationreceiver 3400, the control information generator 3500, the informationprovider 3600, the DB 3700, and the transceiver 3800 in such a way thatthe server 3000 generates the control information for operating thedevice 1000 and the external apparatus 2000 according to the user'sintention based on the user input information, and provides thegenerated control information to the device 1000.

Some or all of the input information receiver 3100, the apparatusinformation receiver 3200, the user intention determiner 3300, thecondition information receiver 3400, the control information generator3500, and the information provider 3600 may be driven by softwaremodules, but are not limited thereto. Some or all of the inputinformation receiver 3100, the apparatus information receiver 3200, theuser intention determiner 3300, the condition information receiver 3400,the control information generator 3500, and the information provider3600 may be hardware devices.

Also, at least some of the input information receiver 3100, theapparatus information receiver 3200, the user intention determiner 3300,the condition information receiver 3400, the control informationgenerator 3500, and the information provider 3600 may be included in thecontroller 3900, and the input information receiver 3100, the apparatusinformation receiver 3200, the user intention determiner 3300, thecondition information receiver 3400, the control information generator3500, the information provider 3600, and the controller 3900 may bedriven by one processor. However, an exemplary embodiment is not limitedthereto.

Exemplary embodiments may be implemented as a medium includingcomputer-executable commands, e.g., a computer-executable programmodule. A computer-readable medium may be an arbitrary medium that maybe accessed by a computer, and may include volatile and nonvolatilemedia, and detachable and non-detachable media. The computer-readablemedium may include a computer recording medium and/or a communicationmedium. The computer recording medium includes volatile and nonvolatilemedia, and detachable and non-detachable media that are embodied byusing an arbitrary method or technology for storing information such asa computer-readable command, a data structure, a program module, orother data. The communication medium may include a computer-readablecommand, a data structure, a program module, or other transmissionmechanisms, and includes an arbitrary information transmission medium.

Exemplary embodiments may, however, be embodied in many different formsand should not be construed as being limited to the embodiments setforth herein; rather, the described exemplary embodiments are providedso that the disclosure will be thorough and complete, and will fullyconvey the inventive concept to those of ordinary skill in the art. Forexample, a single component may be separated into a plurality ofcomponents, while a plurality of components may be combined into onecomponent.

The foregoing exemplary embodiments and advantages are merely exemplaryand are not to be construed as limiting. The present teaching can bereadily applied to other types of apparatuses. The description of theexemplary embodiments is intended to be illustrative, and not to limitthe scope of the claims, and many alternatives, modifications, andvariations will be apparent to those skilled in the art.

What is claimed is:
 1. A mobile device comprising: a memory configuredto store a program; and a processor configured to control one or moreexternal apparatuses, among a plurality of external apparatuses, byexecuting the program, wherein the program comprises commands which,when executed by the processor, cause the processor to: identify the oneor more external apparatuses communicable with and controllable by themobile device, from the plurality of external apparatuses; provideapparatus information of the one or more external apparatuses, to aserver; receive, from the server, control information for controllingthe one or more external apparatuses for a user's intention; andtransmit a control command to the one or more external apparatuses basedon the received control information.
 2. The mobile device of claim 1,the program further comprises commands to transmit user inputinformation to the server, and wherein the user input information isused when the server determines the user's intention.
 3. The mobiledevice of claim 2, wherein the control information is generated, by theserver, based on the user's intention.
 4. The mobile device of claim 2,wherein the user input information is generated by the mobile devicebased on at least one among a text input by a user, a voice of the user,and a gesture of the user.
 5. The mobile device of claim 1, wherein theone or more external apparatuses include a first external apparatuswhich is an external apparatus for checking whether a control conditionaccording to the user's intention is satisfied and a second externalapparatus which is an external apparatus for executing an operationaccording to the user's intention.
 6. The mobile device of claim 1,wherein the received control information is generated, by the server,based on a user input information and the apparatus information.
 7. Themobile device of claim 5, wherein the control condition for controllingthe second external apparatus is determined based on the user'sintention, and the control information is generated based on the controlcondition.
 8. The mobile device of claim 7, wherein the controlinformation comprises information for checking whether the controlcondition is satisfied and information for controlling an operation ofat least one among the mobile device and the second external apparatuswhen the control condition is satisfied.
 9. The mobile device of claim5, wherein the control information comprises a plurality of controlcommands for controlling an operation of at least one among the mobiledevice and the second external apparatus, and the plurality of controlcommands are executed by at least one among the mobile device and thesecond external apparatus in a predetermined order.
 10. A method forcontrolling one or more external apparatuses, the method being performedby a mobile device and comprising: identifying the one or more externalapparatuses communicable with and controllable by the mobile device,from a plurality of external apparatuses; providing apparatusinformation of the one or more external apparatuses, to a server;receiving, from the server, control information for controlling the oneor more external apparatuses for a user's intention; and transmitting acontrol command to the one or more external apparatuses based on thereceived control information.
 11. The method of claim 10, furthercomprising transmitting a user input information to the server, andwherein the user input information is used when the server determinesthe user's intention.
 12. The method of claim 10, wherein the controlinformation is generated, by the server, based on the user's intention.13. The method of claim 11, wherein the user input information isgenerated by the mobile device based on at least one among a text inputby a user, a voice of the user, and a gesture of the user.
 14. Themethod of claim 10, wherein the one or more external apparatuses includea first external apparatus which is an external apparatus for checkingwhether a control condition according to the user's intention issatisfied and a second external apparatus which is an external apparatusfor executing an operation according to the user's intention.
 15. Themethod of claim 10, wherein the received control information isgenerated, by the server, based on a user input information and theapparatus information.
 16. The method of claim 14, wherein the controlcondition for controlling the second external apparatus is determinedbased on the user's intention, and the control information is generatedbased on the control condition.
 17. The method of claim 16, wherein thecontrol information comprises information for checking whether thecontrol condition is satisfied and information for controlling anoperation of at least one among the mobile device and the secondexternal apparatus when the control condition is satisfied.
 18. Themethod of claim 14, wherein the control information comprises aplurality of control commands for controlling an operation of at leastone among the mobile device and the second external apparatus, and theplurality of control commands are executed by at least one among themobile device and the second external apparatus in a predeterminedorder.
 19. A non-transitory computer-readable recording medium havingrecorded thereon a computer program which, when executed by a computersystem, causes the computer system to execute the method of claim 10.